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NOTICES 


This document and the technical information it contains is subject to change by the LiMo Foundation and 
its members at any time without notice as authorized by the LiMo Foundation bylaws and other documents 
governing the LiMo Foundation, including amendments thereto. All statements regarding LiMo’s future 
direction or intent are subject to change or withdrawal without notice, and represent goals and objectives 
only. 


Any reference in this document to non-LiMo Foundation Web sites do not in any manner serve as an 
endorsement of those Web sites. The materials at those Web sites are not part of this document or any 
product described in this document and use of those Web sites is at your own risk. 


Information concerning non-LiMo products was obtained from the suppliers of those products ora public 
announcement or source. LiMo cannot confirm the accuracy of that information or the performance, 
compatibility or any other claims related to non-LiMo products. 


All names or other identifying information used in examples in this document are fictitious and any 
similarity to the names or other identifying information of an actual person or organization is entirely 
coincidental. 


THIS DOCUMENT AND ALL INFORMATION IN THIS DOCUMENT ARE PROVIDED “AS IS”, 
WITHOUT ANY EXPRESS OR IMPLIED WARRANTY OR CONDITION, INCLUDING, BUT NOT 
LIMITED TO, ANY WARRANTY OR CONDITION OF ACCURACY OR COMPLETENESS, 
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE OR AGAINST INFRINGEMENT. 
ALL EXPRESS AND IMPLIED STATUTORY WARRANTIES ARE DISCLAIMED, INCLUDING 
WITHOUT LIMITATION, THE WARRANTY OF MERCHANTABILITY, FITNESS FOR A 
PARTICULAR PURPOSE, TITLE OR NON-INFRINGEMENT. 
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1 Introduction 


The primary audience of this document is for multimedia hardware platform vendors and LiMo multimedia 
framework adapters. It provides the foundation to understand the linkage between the LiMo multimedia 
framework and primarily the OpenMAX Integration Layer. The OpenMAX Integration layer is the chosen 
industry standard hardware abstraction layer for multimedia. Hardware abstraction entities are typically 
provided by silicon vendors. 


This document is divided into three main sections. They are: 


e Ahigh level architecture section that provides a brief description of the LiMo high level 
multimedia architecture and a high level intersection between the framework and [OMX01]. 


e A use case section that defines the typical multimedia use cases that require the hardware 
integration layer services. 


e A section that describes OpenMAX IL, including standard components, that are utilized by the 
LiMo Foundation Platform Release 1 implementation. 


1.1 Purpose 


This document describes the relationship with LiMo multimedia and existing open standards/specifications 
for providing multimedia hardware independence. The three associated specifications are the [LIMO02], 
[LIMO03], and [OMX01]. 


[LIMO03] defines the following three key requirements for the multimedia hardware integration layer: 


e The MM Framework shall support an industry standard hardware abstraction layer so that a wide 
range of hardware devices can be integrated into the multimedia framework with less development 
work. 


e The MM Framework shall minimize and isolate impact to framework as hardware platforms 
evolve. 


e MM framework shall be capable of utilizing HW acceleration including tunneling. 


[LIMO02] defines the multimedia framework component architecture used to realize the LiMo Foundation 
Platform Release 1 use cases. Within the multimedia framework component definition, a multimedia 
hardware integration layer is described. The intent of this layer is to provide a foundation for the LiMo 
multimedia framework component to be developed and easily integrated among different mobile 
multimedia IC solutions. 


[OMX01] is the primary tool for which LiMo multimedia achieves the hardware independence 


requirements. This document will isolate the components and functionality that are applicable to the LiMo 
Foundation Platform Release 1 multimedia architecture and use cases. 


1.2 Scope 
LiMo components are designed independent of a hardware platform, and because in most cases multimedia 
services are closely associated to the hardware, specific hardware integration layer requirements have been 


defined. This document provides guidelines for integrating to the LiMo Hardware Integration Layer as 
defined in the LiMo multimedia framework component. 
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1.3 Definitions, Acronyms and Abbreviations 

1.3.1 Definitions 

OpenMAX IL API: OpenMAX Integration Layer; a streaming media API. 


1.3.2 Acronyms 


A2DP Advanced Audio Distribution Profile (Bluetooth stereo headset profile) 
AAC Advanced Audio Coding (MPEG-2 and MPEG-4) 
AMR-NB Adaptive Multi-Rate (Narrow Band) 

AMR-WB Adaptive Multi-Rate (Wide Band) 

API Application Programming Interface 

DRM Digital Rights Management 

HLA High Level Architecture (LiMo) 

HLR High Level Requirements (LiMo) 

HW Hardware 

IL Integration Layer (OpenMAX) 

MM Multimedia 

MMEW Multimedia Framework 

MP3 MPEG-1 Audio Layer 3 

MPEG Motion Pictures Expert Group 

OMX OpenMAX 

PCM Pulse Code Modulation 

RGB Red, Green, Blue (video color model) 

SBC Sub-Band Codec (Bluetooth stereo headset profile) 
SPEF Security Policy Enforcement Framework 

sw Software 

VE View Finder 

WMA Windows Media Audio 

YUV Luma/Chrominance (video color model) 
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1.4 References 


[LIMO01] 


[LIMO02] 


[LIMO03] 


[OMX01] 


[OMX02] 


High Level Requirements for MMFW, Version 1.0, LiMo Foundation, 1 February 2008 


Software Architecture Document, Version 1.0 Revision 1.6, LiMo Foundation, 22 
October 2007 


High Level Requirements, Version 1.0 Revision 1.5, LiMo Foundation, 22 October 2007 
OpenMAX Integration Layer Application Programming Interface Specification, Version 


1.1, <http://www.khronos.org/files/openmax_il_spec_1_1.pdf>, Khronos Group, 22 
February 2007 


Using OpenMAX Integration Layer with GStreamer, Version 1.0, 
<http://www.khronos.org/openmax/whitepapers/OpenMAX_IL_with_GSstreamer.pdf>, 
ST Microelectronics, 24 April 2006 
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2 LiMo Multimedia Framework 


2.1 High Level Architecture 


The following diagram appears in the LiMo software architecture document and provides a high level view 
of the multimedia framework, From the diagram, it can be seen that the modules directly related to the 
Hardware Integration Layer are the Media Stream Framework (and associated plug-ins) and the Audio 
Framework. 


Application 


Multimedia Framework Ky Telephony 
Framework 


Media Service Framework 


=P) Fromenork 


Audio 
Framework 


Media Stream Framework 


Kem sre 


Hardware Integration Layer 


Figure 1 The Multimedia Framework Block Diagram 


As described in [LIMO02], the Hardware Integration Layer provides hardware independence and is 
especially important in providing an abstraction that shields the multimedia framework from differences in 
mobile multimedia coding environments. 


In LiMo, the hardware integration layer API principally used is that defined in [OMXO01]. While [OMX01] 
provides the bulk of the hardware independence and API framework for LiMo Foundation Platform 
Release 1 requirements, not all accelerated functionality desired by the Hardware Integration Layer utilizes 
[OMX01] APIs. 


Such services are: 
e Audio Device Processing, Rendering, and Path Control 
e LED/Vibrator/Alert control 
e Final Video Rendering 
© Copyright 2007, 2008 LiMo Foundation, Inc. All rights reserved. 
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Such service bindings are not explicitly covered in this document. It is expected that within the LiMo 
Foundation Platform Release 1 time frame such services are platform dependent and will be addressed by 
the system integrator on a per-platform basis. 


2.2 High Level Use of OpenMAX IL in LiMo Foundation Platform Release 1 
Architecture 


This section provides a cross reference to the OpenMAX IL specification [OMX01] and attempts to 
provide a high level description of the functionality and APIs defined within [OMXO1] that are important 
to a LiMo Foundation Platform Release 1 multimedia implementation. 


Section 2.1.1 of [OMX01] provides an architectural overview of the OpenMAX IL. The overview section 
provides a detailed high level description as well as definition of key concepts within OpenMAX IL. It 
should be studied and understood prior to interpretation of this guidelines document. This section of the 
guidelines document will merely cross reference [OMX01] and provide specific comments on key topics of 
specific interest to the LiMo use or non-use of OpenMAX IL. 


2.2.1 Component Profiles 


Section 2.1.3.1 of [OMX01] defines the component profiles that are available in the OpenMAX IL. Two 
profiles are defined; a base profile and an interop profile. One key difference between the base and interop 
profiles is the level of support for tunneled communication between OpenMAX IL components. 


From Table 2-2 in [OMX01], it can be seem that three types of inter-component communication are 
defined; non-tunneled, tunneled, and proprietary. Such communication is used when passing information 
between component input/output ports. Non-tunneled communication defines a mechanism where data 
exchange between components is driven by the client framework. Such mechanisms are not optimal and 
should be avoided in the bulk of the LiMo OpenMAX IL implementation. 


Components are expected to support communication mechanisms that are not required to be driven by the 
source component. Such mechanisms are provided either in the interop profile's tunneled communication 
definition or a base profile's proprietary communication mechanism. The LiMo Foundation Platform 
Release 1 implementation requires interop profile support for Tunneled communication. 


Refer to sections 3.4.1.1 through 3.4.3.2 of [OMX01] for detailed initialization and data flow sequences for 
the various component communication mechanisms. 


2.2.2 Component States 


OpenMAX component states are defined in [OMX01] section 2.1.4. Six states are defined. They are: 


e Invalid 
* Loaded 
e Idle 


e WaitForResources 
e Paused 
° Executing 

The bold states are those readily expected and positively honored by the LiMo Foundation Platform 


Release 1 multimedia framework architecture. The two states, Invalid and WaitForResources are 
negatively honored. They are not expected in normal use case operations of LiMo Foundation Platform 


© Copyright 2007, 2008 LiMo Foundation, Inc. All rights reserved. 
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Release 1 framework architecture. Any deviations must be handled independently during unique system 
integration. 


Refer to [OMX01] section 2.1.4 for detailed definition of the states and transitional aspects of the states. 
2.2.3 Resource Management 


OpenMAX IL resource management is discussed in section 2.1.17 of [OMX01]. OpenMAX IL provides 
the supporting framework for an OpenMAX core implementation to provide complex resource 
management. The infrastructure consists of event and state frameworks that allow the OpenMAX core to 
place components and component groups into "resource pending” type states and notify the client when 
situations dictate such transitions. It also provides the infrastructure to transition and notify when resources 
become available and transition to non "pending" states can be made. Such events are defined in section 
3.1.1.4 of [OMX01] and include: 


e OMX_EventResourcesAcquired 
e OMX_EventComponentResumed 
e OMX_EventDynamicResourcesAvailable 


Resource concealment interfaces and component suspension policies are defined in section 3.1.2.6 of 
[OMX01]. The mechanisms defined in this section of [OMX01] provide the client capabilities to control 
resource suspension policies. 


The LiMo Foundation Platform Release 1 multimedia framework reference implementation does not rely 
on OpenMAX resource management and component suspension policy procedures. A higher level resource 
management framework is available within LiMo. OpenMAX resource related event notifications received 
by the multimedia framework are handled as default negative events and result in the use case being 
denied. 


2.2.4 Client Communication 


The LiMo Foundation Platform Release 1 multimedia framework reference implementation will be that of 
a client that operates in a different context than the OpenMAX core and OpenMAX components. The 
client-OpenMAX core/component communication behavior may resemble either the "Out-of-Context" or 
the "In-Context" descriptions available in [OMX01]; shown in Figure 2-5 and described in section 2.1.6. 
This is implementation dependent. 


2.2.5 Buffer Management 


The LiMo Foundation Platform Release 1 implementation aligns with buffer payloads and data formatting 
as described in [OMX01] section 2.1.21. The OMX_UseBuffer call is shown in the use cases within this 
document. This call requires buffer allocation by the client. For performance purposes, it may be 
advantageous for the OpenMAX IL layer to allocate and manage data buffers. For this, the 
OMX_AllocateBuffer call would be utilized. 


Buffer marking, Content Pipes, and File Parsing are detailed in [OMXO1] sections 2.1.10, 2.1.18, and 
2.1.19 respectively. Such functionality is not required in the LiMo Foundation Platform Release 1 
implementation. 


It should be noted that although not planned for use in the LiMo Foundation Platform Release 1 
implementation, buffer marking could be worth supporting as it may prove useful for performance analysis 


purposes. Use and implementation requirements for buffer management are left to the individual system 
integrators. 
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2.2.6 Clock and Rate Control 


Clock services and rate control are limited in the LiMo Foundation Platform Release 1 implementation. 
Synchronization and Rate Control are described in sections 2.1.14 and 2.1.15 of [OMX01]. 
Synchronization is always driven by audio. In the playback use cases, the audio sink drives the master 
clock and in the capture use cases the audio source drives the master clock. 


Rate control scaling will not be utilized by the LiMo Foundation Platform Release 1 implementation. From 
a rate control scaling perspective, all rates will be hard coded at 1x. Non-play/pause control capabilities 
(e.g. fast forward and rewind) will be provided at the multimedia framework level and are not within the 
LiMo Foundation Platform Release 1 compliant OpenMAX IL implementation. 


2.2.7 OpenMAX Commands 


OpenMAX IL command types are described in Table 3-1 of section 3.1.1.1 of [OMX01]. The bold 
commands identified below are those readily expected to be utilized and positively honored by the LiMo 
LiMo Foundation Platform Release 1 multimedia framework architecture. Specific command use is 
outlined in the use case section of this document. 


e OMX_CommandsStateSet (description in section 3.2.2.3) 
e OMX_CommandFlush (description in section 3.2.2.4) 
e OMX_CommandPortDisable (description in section 3.2.2.5) 
e OMX_CommandPortEnable (description in section 3.2.2.6) 
e OMX_CommandMarkBuffer (description in section 3.2.2.7) 


2.2.8 Component Defi! 


jons 


Refer to section 4 on page 7. 
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3 Use Cases 


This section describes the interactions with the OpenMAX IL layers for the typical LiMo Foundation 
Platform Release 1 use cases. It provides high level ladder diagrams for the use cases as well as 
breakdowns of key interfaces, components, and important details specific to the use case. 


Tt should be noted that these descriptions are representative in nature. The uses cases, ladder diagrams, and 
underlying descriptions are neither exhaustive nor definitive. 


3.1 OpenMAX IL and Hardware Abstraction Initialization 

As described in section 3.2.3.1 of [OMX01], OMX_Init is the first call made to the OpenMAX IL layer. It 
is a OpenMAX core method called to initialize the OpenMAX core. It is executed prior to or at the start of 
use case realization and results in the initialization of the OpenMAX IL core. 


At initialization, all components are recognized and available to the client. The LiMo Foundation Platform 
Release 1 implementation does NOT support dynamic installation of new components. After initialization, 
a detailed list of components available in the system can be obtained via the OMX_ComponentNameEnum 
method. Specific component support is integrator implementation dependent, but will largely follow the 
component definitions available in the component portion [4] of this document. 


Handles to individual components can be obtained via the OMX_GetHandle method and released via the 
OMX_FreeHandle method. 


OMX_Deinit must be called before another initialization sequence may be initiated. 


3.2 Audio/Video Playback 


The Audio/Video Playback use case consists of a series decoded audio and video data frames. The frames 
of the encoded bit stream data are delivered by the multimedia framework at a rate that achieves a 
continuous stream of output data. Figure 2 and Figure 3 provide graphical representations of the 
audio/video playback use case. Figure 2 provides a component diagram similar to what is expected in a full 
HW pipeline implementation. Figure 3 provides a component diagram for a solution that is not fully 
hardware pipelined. In the Figure 3 example, it is likely that only the decode portion is accelerated in 
hardware and some level of rendering is required in software. 


In the HW pipeline representation of Figure 2, the video components consist of the video decoder, video 
post processor, video scheduler, and video renderer. The video post processor is needed to coordinate post 
processing activities that may not be present in all decoders (e.g. deblocking). 


The audio components consist of the audio decoder, audio post processor, and audio renderer. 


The clock component utilizes audio as a reference and provides the synchronization needed in rendering of 
decompressed audio and video data. 


© Copyright 2007, 2008 LiMo Foundation, Inc. All rights reserved. 
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Figure 2 HW Pipelined Audio/Video Playback OpenMAX IL Component Group 


In the partial SW pipeline representation of Figure 3, the video components consist of the video decoder, 
video post processor, and video scheduler. The output of the video scheduler is provided to the client 
framework when audio and video are scheduled to be rendered. 


The audio components consist of the audio decoder, audio post processor, and audio renderer. An audio 
renderer component is shown in this example. In this example the component is tightly coupled with the 
client OpenMAX frameworks since it provides the master clock for the use case. 


From Figure 2 and Figure 3, it can be seen that demux is not within the scope of the OpenMAX IL 
implementation. 


In all cases, the audio and video decoders are the initial components utilized in the OpenMAX IL 
component group and tunneled communication is utilized between components until the audio rendering 
and video scheduling/rendering. The reference framework will provide the logic to define the audio/video 
use case and initialize and operate the appropriate component group to realize the use case. It is up to the 
integrator to make the binding between the audio and video renderers and the independent framework that 
the integrator utilizes. 
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Figure 3 Partial SW Audio/Video Playback OpenMAX IL Component Group 


Figure 4 provides a sequence of OpenMAX IL methods that are utilized to realize the HW pipelined 
audio/video playback use case shown in Figure 2. As previously noted, the sequence is neither exhaustive 
nor definitive. The key items to note from the sequence diagram are that individual component 
communication between the client and core generally only occurs in the LOADED state. Once the 
appropriate tunnels have been established and all components are in the IDLE and EXECUTING states, 
buffer management operations are performed on the ports that the client has direct access to. 


In this sequence, the client allocates buffers and passes buffer details to the audio and video decode input 
ports. The decode-to-rendering sequence is directed by the client by instructing the decoders to empty the 
buffers that the de-mux provides. Rendering timing is also provided. Once the content is decoded, post 
processed, scheduled, and finally rendered, the client receives notice from the rendering component that the 
buffer has been emptied. The process of receiving and processing buffers at the downstream end of the 
pipeline and subsequently passing the empty buffers back to upstream components drives the decode-to- 
rendering iterations. This sequence is shown in finer detail in section 3.5 when isolated to an audio 
playback use case. 


In non-HW pipeline implementations where rendering is coupled to the client framework such as that 


shown in Figure 3, buffers would be supplied by the client framework at the video scheduler and filled as 
video rendering is scheduled. 
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Figure 4 Audio/Video Playback OpenMAX IL Message Sequence Chart 


It should be noted that dynamic port reconfiguration should be supported by both the client and the 
OpenMAX IL implementation. Assuming a default port setting mismatch, it is expected that dynamic port 
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reconfiguration could appear at the video decoder input port once the video configuration information is 
processed. The client should be able to handle this type of operation. 


Dynamic port reconfiguration is described in section 3.4.5 of [OMX01]. A typical example is described and 
a detailed message sequence chart is also provided in Figure 3-18 of the same reference. In this example a 
port mismatch is after the in-band video configuration information is parsed by the decoder components 
resulting in a mismatch against the default port configuration. One such example would be a different 
source video resolution. It is beneficial for OpenMAX IL implementations and integrated LiMo multimedia 
frameworks to support such operations. 


3.3 Audio/Video Playback Control 


The Audio/Video Playback control use case outlines the procedures to pause an audio/playback use case, 
and seek to a new position (e.g. seek forward or seek backward) in the content prior to resuming the use 
case. The seek itself is performed within the Media Stream Framework. The HW pipelined component 
group is the same as identified in Figure 2. Figure 5 provides a description of the event sequence. 


One thing that is not shown in any of the sequences thus far is the use of multiple OpenMAX buffers. The 
sequence diagrams only show a single buffer per multimedia thread; however, implementations must 
support multiple buffers. The framework will feed the OpenMAX IL decoder(s) until throttled/queued off. 
It is expected that minimally, two frames of data may be queued up for rendering at any point in time. 


Section 3.2.2.4 of [OMX01] defines the flush procedures in detail. From Figure 5, it should be noticed that 
once the component group is transitioned to the pause state, the flush command is issued on the ports that 
the client has direct control. This is because the component group is setup for tunneling. The decode 
components return the buffers to the client, and once returned, the group may be transitioned back to the 
executing state. This concludes the operation and new data is submitted for decode and rendering. Please 
note that a flush is not required for a simple pause/resume (non-seek) instance of this control sequence. 


It is key to note that the bulk of the seek operation is performed outside of the scope of the OpenMAX IL 
implementation. As noted in section 2.2.6, rate control scaling services are not required to be supported of 
the OpenMAX IL implementation. This means that non-play/pause operations (e.g. fast forward and 
rewind) are to be performed outside of the OpenMAX IL implementation and within the LiMo Foundation 
Platform Release 1 implementations. 
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Figure 5 Audio/Video Seek OpenMAX IL Message Sequence Chart 


3.4 Image Capture 


Figure 6 provides a description of the component group required for the image capture use case. The 
camera component is utilized to provide a view finder output and outputs for a thumbnail capture and a full 
size image capture. 


It should be noted that the camera component defined is different than the standard camera component 
defined in section 8.9.1 of [OMX01]. The standard component is limited to a single image output port. The 
component shown in Figure 6 has two image output ports; one for the full captured image and another for a 
thumbnail representation of the captured image. While the thumbnail port is optional for the LiMo 
Foundation Platform Release 1 implementation, its availability simplifies the media stream framework 
design and is strongly recommended. 
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Figure 6 Still Image Preview/Capture OpenMAX IL Component Group 


The mux entity in the diagram represents the framework's requirements to coordinate combining of full 
image, metadata, and thumbnail into the appropriate file format (e.g. JFIF or EXIF). 


Figure 7 provides a description of the message sequence chart for the image capture use case. From the 
message sequence chart, it can be seen that port disable/enable functionality may be utilized while the view 
finder is configured and operational. When the framework triggers the image capture, the image encode 
ports are enabled. This is shown because some client frameworks may not distinguish between image or 
video capture prior to the client triggering the "start" of the capture operation. Such support is not required 
by the LiMo Foundation Platform Release 1 implementation, but suggested. 


This example does differ from the still image capture example described in section 8.9.1.3 of [OMXO1]. As 
described in the OpenMAX example, all ports are defined and enabled when the component group is 
defined. The capture control bit is used to trigger the capture and causes data for a single image to be 
emitted on the image capture port(s). This is also expected to be a reasonable client implementation and 
must be supported. 
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397 Figure 7 Image Capture OpenMAX IL Message Sequence Chart 


398 3.5 Audio Playback 


399 The Audio Playback use case is very similar to the audio portion of the Audio/Video playback group as 
400 described in section 3.2. The main difference is that it does NOT include the video chain or the clock 
401 component. 


402 Figure 8 provides a description of the component group required for the audio playback use case. As in the 
403 Audio/Video playback case, the file de-mux operation occurs within the multimedia framework. The HW 
404 pipelined group consists of an audio decoder, audio post processor, and audio renderer. A single audio post 
405 processing component is shown here mainly as a place holder to indicate where advanced audio post 
406 processing algorithms could be included by the system integrator. Although system integrators may provide 
407 differing volume control architectures, it should be noted that per-stream volume control is provided by the 
408 audio renderer component in alignment with the standard component definitions of [OMX01]. 
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A couple of key items to note from audio that also relate to the Audio/Video combination is that audio is 
managed as a single stream and PCM is the only supported output of the OpenMAX component group. 
This means that advanced functionality such as mixing and rendering to non-PCM devices (e.g. A2DP 
profile for Bluetooth stereo headset) must be covered independently in the system integrator's audio 
framework. 


> Media Stream Framework 
filemp3 pe = 
| Framework-OMX Audio Playback Use Case Binding | 
Due 3 
OMX Il» — Rende 
sv Aud Aud Aud To Mixer/ Audio 
Provided — oee HO Pe ~~~ Rea Service 


Figure 8 Audio Playback OpenMAX IL Component Group (HW Pipelined) 


The message sequence for the Audio Playback case is very similar to the Audio/Video playback case, thus 
it is not described here. What is shown here in Figure 9 is a simple flow of the buffering/rendering of the 
audio data. 


The key items to recognize from the diagram are that the buffers are supplied to the input port of the audio 
decoder (not explicitly shown here, but outlined in the message sequence chart of section 3.2) and the audio 
renderer drives the overall "pull" process via rendering at a constant bit rate or utilizing timing information 
within the buffer itself. 


In addition, as previously noted, a full HW pipelined solution may not be implemented in all silicon and 
software platforms. In such cases, it is expected that buffers supplied to the audio post processor are filled 


and provided back to the client framework. In this case, the client framework drives the "pull" operation 
from the output of the audio post processor component. 
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Figure 9 Audio Buffering/Rendering Message Sequence Chart (HW Pipelined) 


From the diagram it should be noted that three buffers are shown at the audio decoder and two at the audio 
render. The intention is to show that the operations on such buffers are only indirectly related. As 
previously discussed, it is the emptying of the buffers by the audio renderer that drives the timed delivery 
of the synchronous audio data stream. 


The communication between components within the playback chain is shown as dotted lines. This is of 
intention to make the point that differences may exist between OpenMAX IL implementations and some 
inter-component communication may be proprietary in nature. 


3.6 Audio/Video Capture 


Figure 10 provides a description of the HW pipelined audio/video capture component group. The 
audio/video capture sequence builds upon the other use cases whose description has already been provided. 
Of interest to this use case are the fact that the camera component provides multiple video output ports; one 
for the view finder (e.g. may be RGB or YUV format dependent on the video rendering system), and 
another of YUV format as input to the video encoder component. The outputs of the video encoder and 
audio encoder are provided to the multimedia framework and the audio/video information is multiplexed 
and formatted outside of the OpenMAX IL implementation. 
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Figure 10 Audio/Video Capture OpenMAX IL Component Group 


The clock component, which must be updated (set to zero) by the client at start, may be utilized by the 
camera component and audio capture components to provide the same reference timing for encoded audio 
and video data. The timestamp is tagged in the capture buffers at the specific time of capture and may be 
utilized during the encode process as well as within the client for audio/video synchronization. 
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452 4 Standard Component Use 


453 As described in section 8 of [OMX01], OpenMAX IL provides a list and definition of standard components 
454 to facilitate component portability. Table 1 outlines the subset of the standard component use within the 
455 reference multimedia implementation for LiMo Foundation Platform Release 1. 


456 The intent of this list is not for an exhaustive list of required components, but merely a consolidated list of 
457 component class functionality required to meet the hardware independent layer. It is expected that 
458 integrators and silicon vendors may wish to extend this in a commercial implementation of a LiMo 
459 Foundation platform solution. 


460 Table 1 OpenMAX IL Standard Components Utilized by the MMFW 
Type Class Component Ports Control/Parameters 
ere re Compressed Input, 
Output 
oas | WBE Input, Compressed 
Output 
Image PNG, BMP, 
Decoder WBMP, GIF, Input, YUV Output 
Animated GIF 
Renderer | YUV/RGB Input Resolution 
Overlay 
H.263 
AVC (H.264) 


Resolution, frame rate, bit rate, 


Compressed Input, | compression format, profile, output 


Decoders [MEEG4 YUV/RGB Output 


Real Video format 
WMY 
H.263 - 
ee fuse Resolution, frame rate, bit rate, 
Video | Encoder | AVC (H.264) Carendi Outpt compression format, profile, input 
MPEG-4 format 


Video Input, Video 


Scheduler 7 Output, Clock Input 


Video Input, Video | E.G. control conversion from YUV 
Processor | Misc. 


Output to RGB format and vice versa 
Renderer | YUV/RGB Input Resolution 
Overlay 
Clock Input, Output | Multiple image capture associated 
Camera | Camera 2 parameters such as zoom, white 


(VF), Output (ENC) balance, etc... 


(Continued on next page.) 
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463 
Type | Class Component Ports Control/Parameters 
AME Sample rate, mode type, payload 
format 
ANENE Sample rate, mode type, payload 
format 
Channels, sample rate, bit rate, 
Deea MP3 Compressed Input, | format 
PCM Output 
Channels, sample rate, bit rate, 
AAC : 
profile, stream format 
Sin Channels, sample rate, bit rate, 
format 
REAL AUDIO Channels, sample rate, bit rate 
Audio ANNB Sample rate, mode type, payload 
format 
a GGaeen PCM Input, Sample rate, mode type, payload 
Compressed Output | format 
re Channels, sample rate, bit rate, 
profile, stream format 
: PCM Input, PCM | Audio processing algorithms are 
Processor | X (Misc.) Output implementation dependent 
Renderer | PCM Renderer PCM Input, Clock | Mute and volume 
Input 
Capturer | PCM Capturer Clock Input, PCM | Channels and sample rate 
Output 
Syne | Other Clock Output(s) Implementation dependent 
464 
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